Unsupervised Learning Capstone¶
# importing packages we'll need
import numpy as np
import pandas as pd
import scipy
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
from sklearn.linear_model import LinearRegression, LassoCV, RidgeCV, ElasticNetCV
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_absolute_error
from statsmodels.tools.eval_measures import mse, rmse
from sklearn.linear_model import Ridge
from sklearn.linear_model import Lasso
from sklearn.linear_model import ElasticNet
from sklearn import linear_model
import statsmodels.api as sm
from sklearn.svm import SVR
from sklearn.model_selection import cross_val_score
from sklearn import tree
from sklearn import ensemble
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
import warnings
warnings.filterwarnings('ignore')
# reading the data set
df = pd.read_csv('loan.csv', dtype = {'issue_d': str}, low_memory=False)
# https://www.kaggle.com/wendykan/lending-club-loan-data
Data Exploration¶
In order to begin to understand our data, let's take a look at the summary statistics for the dollar amounts of these loans.
df.loan_amnt.describe()
# getting summary statistics
The first thing to note is that Lending Club facilitated an amazing 1,048,575 loans over this period! Apparently a lot of people are comfortable investing in these loans. It appears that the range of allowable loan sizes is \$1,000-\$40,000. The mean loan size of \$15,385 looks to be representative of a typical loan based on the 25th, 50th, and 75th percentile values. It looks like loan sizes are more common towards the lower end of the acceptable range, but let's check for sure.
# creating a histogram
plt.hist(df.loan_amnt)
plt.title('Loan Amount Distribution')
plt.ylabel('Count')
plt.xlabel('Loan Amount, $')
plt.show()
We were right, there are definitely more loans toward the lower end of the range. In fact, it looks as though loan sizes follow a gamma distribution. Let's construct a q-q plot to see if this is, in fact, a gamma distribution.
q = np.random.gamma(5, 1, 1048575) # generating a random gamma distribution
q10000 = np.random.choice(q, 10000) # taking 10000 instances
q10000.sort() # sorting for q-q plot
loan_amts = list(df.loan_amnt)
loan_amts10000 = np.random.choice(loan_amts, 10000) # taking 10000 instances
loan_amts10000.sort()# sorting for q-q plot
plt.scatter(x=q10000, y=loan_amts10000)
plt.ylabel('Loan Amounts')
plt.xlabel('Random Gamma Variable')
plt.title('Q-Q Plot, Loan Amounts and Gamma Distribution')
plt.show()
It looks like much of the data follows a rough gamma distribution, except the tail is too fat to truly be considered a gamma distribution.
The whole point of investing is to get a return. Are the returns for investors worthwhile?
df.int_rate.describe()
The interest rates on these loans range from 5.31% to an eye-popping 30.99%. It looks like there is potential to make some serious cash here. But wait, what if the rate of 30.99% is an extreme outlier? Let's take a look at the distibution to get a better idea of what to expect.
plt.hist(df.int_rate)
plt.title('Interest Rate Distribution')
plt.ylabel('Count')
plt.xlabel('Interest Rate, %')
plt.show()
Great! Although we ought not expect a 30% return, it looks like double-digit returns are very common. Finally, let's see how long our money would be tied up in one of these loans.
plt.figure(figsize=(10, 5)) #making a subplot
plt.subplot(1, 2, 1)
df.term.value_counts().plot(kind='bar')
plt.title('Loan Terms')
plt.ylabel('Count')
plt.xlabel('Term')
plt.subplot(1, 2, 2)
df.term.value_counts().plot(kind='pie')
plt.title('Loan Terms')
plt.legend()
plt.show()
plt.figure(figsize=(18, 7))
plt.subplot(1, 2, 1)
grades = df.groupby('grade') # grouping the data by loan grade
grades.int_rate.mean().plot(kind='bar')
plt.title('Average Interest Rate by Grade')
plt.ylabel('Average Interest Rate')
plt.xlabel('Grade')
plt.subplot(1, 2, 2)
subgrades = df.groupby('sub_grade') # grouping the data by sub-grade
subgrades.int_rate.mean().plot(kind='bar')
plt.title('Average Interest Rate by Subgrade')
plt.ylabel('Average Interest Rate')
plt.xlabel('Subgrade')
plt.show()
plt.hist(df.all_util, bins=10, range=(0, 100))
plt.title('Total Utilization Rate for All Borrowers')
plt.ylabel('Count')
plt.xlabel('Total Utilization Rate')
plt.show()
plt.hist(df.dti, bins=10, range=(0, 45))
plt.title('DTI Ratio for All Borrowers')
plt.ylabel('Count')
plt.xlabel('Debt to Income Ratio')
plt.show()
df.info()
We can see that there are 145 columns of data and over 1 million rows, each representing a unique loan. The list of columns is mostly self-explanatory. We can see that 36 of these columns are strings, which means they are likely to be categorical values. The remaining 109 columns are either floats or integers, likely suggesting continuous variables.
df.nunique()
It's interesting to note that there are over 1,500 unique loan amounts, but over 69,000 unique interest rates. The grades (7) and subgrades (35) will probably be the most important categorical features in our model. We can also see that certain columns, such as 'url', are likely to be unimportant since they have 1 or 0 unique values.
df.isnull().sum()*100/df.isnull().count()
Here we can see what percentage of entries for any given column are null. We will use this information in the data cleaning process prior to building any models.
Next, we'll take a look at how the numerical columns correlate with our target, the interest rate.
# filtering for numerical columns only
df_num = df.select_dtypes(exclude=['object'])
df_num.head()
corr_list = []
for i in col_names:
corr_list.append(df_num[i].corr(df_num['int_rate']))
new_df = pd.DataFrame()
new_df['term'] = col_names
new_df['corr'] = corr_list
new_df.head()
new_df.sort_values(by=['corr'], ascending=False)
Now this list will help inform our decisions about which columns/features are likely to be useful in predicting interest rate.
Finally, let's take a look at the list of columns with data type string so that we can get a good idea of what categorical values will be available as potential features.
df_obj = df.select_dtypes(include=['object'])
list(df_obj)
Data Cleaning and Feature Selection¶
# creating the manual feature list
manual_feat_list = ['int_rate', 'all_util', 'percent_bc_gt_75', 'dti_joint', 'num_tl_op_past_12m', 'inq_last_6mths', 'open_il_12m', 'bc_open_to_buy', 'total_bc_limit', 'mo_sin_old_rev_tl_op', 'sec_app_mort_acc', 'annual_inc_joint', 'pub_rec_bankruptcies', 'term', 'sub_grade', 'home_ownership']
# creating a dataframe with only the selected manual features
man_df = pd.DataFrame()
man_df = df.filter(items=manual_feat_list)
man_df.head()
Now we will get dummy values for the categorical features in the manual feature list.
man_df = pd.concat([man_df, pd.get_dummies(man_df['term'], drop_first=True)], axis=1)
# dropping the string columns now that we've gotten dummies
man_df = man_df.drop('sub_grade', 1)
man_df = man_df.drop('home_ownership', 1)
man_df = man_df.drop('term', 1)
man_df.head()
man_df.fillna(0, inplace=True)
Finally, we replace all NaN values with zeroes in the manual features dataframe. Our manual features are now ready to feed into a model.
Clusters¶
trim_df = man_df.sample(10000, random_state=42)
trim_df.head(10)
trim_df2 = trim_df.copy()
Mean Shift¶
We'll begin with Mean Shift as our first clustering technique. We will use SK Learn's estimate_bandwidth function and see how many clusters are identified "out of the box."
from sklearn.cluster import MeanShift, estimate_bandwidth
# Here we set the bandwidth. This function automatically derives a bandwidth
# number based on an inspection of the distances among points in the data.
my_bandwidth = estimate_bandwidth(trim_df)
my_bandwidth
# Declare and fit the model.
ms = MeanShift(bandwidth=my_bandwidth, bin_seeding=True)
ms.fit(trim_df)
# Extract cluster assignments for each data point.
labels = ms.labels_
# Coordinates of the cluster centers.
cluster_centers = ms.cluster_centers_
# Count our clusters.
n_clusters_ = len(np.unique(labels))
print("Number of estimated clusters: {}".format(n_clusters_))
trim_df['cluster'] = labels
trim_df.cluster.value_counts()
plt.figure(figsize = (15,12))
sns.scatterplot(y='int_rate', x='all_util', hue='cluster', data=trim_df, palette='Spectral')
plt.ylabel('Interest Rate')
plt.xlabel('Utilization Rate')
plt.title('Interest Rate vs Utilization Rate, Colored by Cluster')
plt.show()
plt.figure(figsize = (15,12))
sns.scatterplot(y='int_rate', x='dti_joint', hue='cluster', data=trim_df, palette='Spectral')
plt.ylabel('Interest Rate')
plt.xlabel('Debt to Income Ratio')
plt.title('Interest Rate vs DTI Ratio, Colored by Cluster')
plt.show()
plt.figure(figsize = (15,12))
sns.scatterplot(y='int_rate', x='bc_open_to_buy', hue='cluster', data=trim_df, palette='Spectral')
plt.ylabel('Interest Rate')
plt.xlabel('Bank Card Available Credit')
plt.title('Interest Rate vs Available Bank Card Credit, Colored by Cluster')
plt.show()
plt.figure(figsize = (15,12))
sns.scatterplot(y='int_rate', x='mo_sin_old_rev_tl_op', hue='cluster', data=trim_df, palette='Spectral')
plt.ylabel('Interest Rate')
plt.xlabel('Months Since Oldest Credit Line Opened')
plt.title('Interest Rate vs Months Since Oldest Credit Line Opened, Colored by Cluster')
plt.show()
plt.figure(figsize = (15,12))
sns.scatterplot(y='int_rate', x='total_bc_limit', hue='cluster', data=trim_df, palette='Spectral')
plt.ylabel('Interest Rate')
plt.xlabel('Total Bank Card Limit')
plt.title('Interest Rate vs Total Bank Card Limit, Colored by Cluster')
plt.show()
plt.figure(figsize = (15,12))
sns.scatterplot(y='annual_inc_joint', x='bc_open_to_buy', hue='cluster', data=trim_df, palette='Spectral')
plt.ylabel('Annual Income')
plt.xlabel('Bank Card Available Credit')
plt.title('Annual Income vs Available Bank Card Credit, Colored by Cluster')
plt.show()
34 clusters seems like a lot for this data set, since several clusters have only 1-3 data points. Also, from our diagnostic plots, it looks like the clustering algorithm is keying off of available credit metrics, but not much else is clear.
As our final piece of analysis for this clustering technique, we will take a random sample of the data a re-run the clustering algorithm. We'll then feed the cluster assignments from each run into the Adjusted Rand Index function to get an idea of how stable this particular technique is.
xtra_trim_df = trim_df.sample(2000, random_state=42)
temp_list = xtra_trim_df['cluster']
xtra_trim_df = xtra_trim_df.drop('cluster', 1)
predict = ms.fit_predict(xtra_trim_df)
from sklearn import metrics
metrics.adjusted_rand_score(temp_list, predict)
xtra_trim_df = trim_df.sample(2000, random_state=20)
temp_list = xtra_trim_df['cluster']
xtra_trim_df = xtra_trim_df.drop('cluster', 1)
predict = ms.fit_predict(xtra_trim_df)
from sklearn import metrics
metrics.adjusted_rand_score(temp_list, predict)
xtra_trim_df = trim_df.sample(2000, random_state=2)
temp_list = xtra_trim_df['cluster']
xtra_trim_df = xtra_trim_df.drop('cluster', 1)
predict = ms.fit_predict(xtra_trim_df)
from sklearn import metrics
metrics.adjusted_rand_score(temp_list, predict)
The average ARI is 0.871 and the range of outcomes is 0.085. We'll have to do the same analysis on several other clustering techniques to see how this technique compares.
Mean Shift 2¶
Since I felt like the first iteration of Mean Shift identified too many clusters, this time we will manually set the bandwidth and come up with only 5 clusters.
ms0 = MeanShift(bandwidth=75000, bin_seeding=True)
ms0.fit(trim_df0)
# Extract cluster assignments for each data point.
labels0 = ms0.labels_
# Coordinates of the cluster centers.
cluster_centers0 = ms0.cluster_centers_
# Count our clusters.
n_clusters_0 = len(np.unique(labels0))
print("Number of estimated clusters: {}".format(n_clusters_0))
trim_df0.cluster.value_counts()
plt.figure(figsize = (15,12))
sns.scatterplot(y='int_rate', x='all_util', hue='cluster', data=trim_df0, palette='Spectral')
plt.ylabel('Interest Rate')
plt.xlabel('Utilization Rate')
plt.title('Interest Rate vs Utilization Rate, Colored by Cluster')
plt.show()
plt.figure(figsize = (15,12))
sns.scatterplot(y='int_rate', x='dti_joint', hue='cluster', data=trim_df0, palette='Spectral')
plt.ylabel('Interest Rate')
plt.xlabel('Debt to Income Ratio')
plt.title('Interest Rate vs DTI Ratio, Colored by Cluster')
plt.show()
plt.figure(figsize = (15,12))
sns.scatterplot(y='int_rate', x='bc_open_to_buy', hue='cluster', data=trim_df0, palette='Spectral')
plt.ylabel('Interest Rate')
plt.xlabel('Bank Card Available Credit')
plt.title('Interest Rate vs Available Bank Card Credit, Colored by Cluster')
plt.show()
plt.figure(figsize = (15,12))
sns.scatterplot(y='int_rate', x='mo_sin_old_rev_tl_op', hue='cluster', data=trim_df0, palette='Spectral')
plt.ylabel('Interest Rate')
plt.xlabel('Months Since Oldest Credit Line Opened')
plt.title('Interest Rate vs Months Since Oldest Credit Line Opened, Colored by Cluster')
plt.show()
plt.figure(figsize = (15,12))
sns.scatterplot(y='int_rate', x='total_bc_limit', hue='cluster', data=trim_df0, palette='Spectral')
plt.ylabel('Interest Rate')
plt.xlabel('Total Bank Card Limit')
plt.title('Interest Rate vs Total Bank Card Limit, Colored by Cluster')
plt.show()
plt.figure(figsize = (15,12))
sns.scatterplot(y='annual_inc_joint', x='bc_open_to_buy', hue='cluster', data=trim_df0, palette='Spectral')
plt.ylabel('Annual Income')
plt.xlabel('Bank Card Available Credit')
plt.title('Annual Income vs Available Bank Card Credit, Colored by Cluster')
plt.show()
From the diagnostic plots it looks like people with little or no debt and with relatively lower incomes are being assigned to cluster 0. Annual income, debt to income ratio, and available credit all appear to have strong predictive power for cluster assignment. There is much more observable differentiation amongst clusters in this iteration than in the previous.
xtra_trim_df0 = trim_df0.sample(2000, random_state=42)
temp_list0 = xtra_trim_df0['cluster']
xtra_trim_df0 = xtra_trim_df0.drop('cluster', 1)
predict0 = ms0.fit_predict(xtra_trim_df0)
metrics.adjusted_rand_score(temp_list0, predict0)
xtra_trim_df0 = trim_df0.sample(2000, random_state=20)
temp_list0 = xtra_trim_df0['cluster']
xtra_trim_df0 = xtra_trim_df0.drop('cluster', 1)
predict0 = ms0.fit_predict(xtra_trim_df0)
metrics.adjusted_rand_score(temp_list0, predict0)
xtra_trim_df0 = trim_df0.sample(2000, random_state=2)
temp_list0 = xtra_trim_df0['cluster']
xtra_trim_df0 = xtra_trim_df0.drop('cluster', 1)
predict0 = ms0.fit_predict(xtra_trim_df0)
metrics.adjusted_rand_score(temp_list0, predict0)
The average ARI is 0.930 and the range is 0.057. This iteration seems to produce a more stable cluster assignment than the first iteration.
Now we'll take a look at K-Mean clustering.
K-Means, K=5¶
Since we are forced to pick the number of clusters with K-Means and since 5 clusters performed best in Mean-Shift, we will start with K=5.
from sklearn.cluster import KMeans
km2 = KMeans(n_clusters=5, random_state=42).fit(trim_df2)
km2_labels = km2.labels_
km2_cluster_centers = km2.cluster_centers_
trim_df2.cluster.value_counts()
plt.figure(figsize = (15,12))
sns.scatterplot(y='int_rate', x='all_util', hue='cluster', data=trim_df2, palette='Spectral')
plt.ylabel('Interest Rate')
plt.xlabel('Utilization Rate')
plt.title('Interest Rate vs Utilization Rate, Colored by Cluster')
plt.show()
plt.figure(figsize = (15,12))
sns.scatterplot(y='int_rate', x='dti_joint', hue='cluster', data=trim_df2, palette='Spectral')
plt.ylabel('Interest Rate')
plt.xlabel('Debt to Income Ratio')
plt.title('Interest Rate vs DTI Ratio, Colored by Cluster')
plt.show()
plt.figure(figsize = (15,12))
sns.scatterplot(y='int_rate', x='bc_open_to_buy', hue='cluster', data=trim_df2, palette='Spectral')
plt.ylabel('Interest Rate')
plt.xlabel('Bank Card Available Credit')
plt.title('Interest Rate vs Available Bank Card Credit, Colored by Cluster')
plt.show()
plt.figure(figsize = (15,12))
sns.scatterplot(y='int_rate', x='mo_sin_old_rev_tl_op', hue='cluster', data=trim_df2, palette='Spectral')
plt.ylabel('Interest Rate')
plt.xlabel('Months Since Oldest Credit Line Opened')
plt.title('Interest Rate vs Months Since Oldest Credit Line Opened, Colored by Cluster')
plt.show()
plt.figure(figsize = (15,12))
sns.scatterplot(y='int_rate', x='total_bc_limit', hue='cluster', data=trim_df2, palette='Spectral')
plt.ylabel('Interest Rate')
plt.xlabel('Total Bank Card Limit')
plt.title('Interest Rate vs Total Bank Card Limit, Colored by Cluster')
plt.show()
plt.figure(figsize = (15,12))
sns.scatterplot(y='annual_inc_joint', x='bc_open_to_buy', hue='cluster', data=trim_df2, palette='Spectral')
plt.ylabel('Annual Income')
plt.xlabel('Bank Card Available Credit')
plt.title('Annual Income vs Available Bank Card Credit, Colored by Cluster')
plt.show()
Available credit and annual income seem to be important variables for this clustering assignment, as we see clear differentiation when those variables are plotted.
xtra_trim_df2 = trim_df2.sample(2000, random_state=42)
temp_list2 = xtra_trim_df2['cluster']
xtra_trim_df2 = xtra_trim_df2.drop('cluster', 1)
predict2 = km2.fit_predict(xtra_trim_df2)
metrics.adjusted_rand_score(temp_list2, predict2)
xtra_trim_df2 = trim_df2.sample(2000, random_state=20)
temp_list2 = xtra_trim_df2['cluster']
xtra_trim_df2 = xtra_trim_df2.drop('cluster', 1)
predict2 = km2.fit_predict(xtra_trim_df2)
metrics.adjusted_rand_score(temp_list2, predict2)
xtra_trim_df2 = trim_df2.sample(2000, random_state=2)
temp_list2 = xtra_trim_df2['cluster']
xtra_trim_df2 = xtra_trim_df2.drop('cluster', 1)
predict2 = km2.fit_predict(xtra_trim_df2)
metrics.adjusted_rand_score(temp_list2, predict2)
The average ARI is 0.976 and the range is 0.018. This is clearly the best performance so far.
Now let's increase the number of clusters and see how performance changes.
K-Means, K=12¶
from sklearn.cluster import KMeans
km3 = KMeans(n_clusters=12, random_state=42).fit(trim_df3)
km3_labels = km3.labels_
km3_cluster_centers = km3.cluster_centers_
trim_df3.cluster.value_counts()
plt.figure(figsize = (15,12))
sns.scatterplot(y='int_rate', x='all_util', hue='cluster', data=trim_df3, palette='Spectral')
plt.ylabel('Interest Rate')
plt.xlabel('Utilization Rate')
plt.title('Interest Rate vs Utilization Rate, Colored by Cluster')
plt.show()
plt.figure(figsize = (15,12))
sns.scatterplot(y='int_rate', x='dti_joint', hue='cluster', data=trim_df3, palette='Spectral')
plt.ylabel('Interest Rate')
plt.xlabel('Debt to Income Ratio')
plt.title('Interest Rate vs DTI Ratio, Colored by Cluster')
plt.show()
plt.figure(figsize = (15,12))
sns.scatterplot(y='int_rate', x='bc_open_to_buy', hue='cluster', data=trim_df3, palette='Spectral')
plt.ylabel('Interest Rate')
plt.xlabel('Bank Card Available Credit')
plt.title('Interest Rate vs Available Bank Card Credit, Colored by Cluster')
plt.show()
plt.figure(figsize = (15,12))
sns.scatterplot(y='int_rate', x='mo_sin_old_rev_tl_op', hue='cluster', data=trim_df3, palette='Spectral')
plt.ylabel('Interest Rate')
plt.xlabel('Months Since Oldest Credit Line Opened')
plt.title('Interest Rate vs Months Since Oldest Credit Line Opened, Colored by Cluster')
plt.show()
plt.figure(figsize = (15,12))
sns.scatterplot(y='int_rate', x='total_bc_limit', hue='cluster', data=trim_df3, palette='Spectral')
plt.ylabel('Interest Rate')
plt.xlabel('Total Bank Card Limit')
plt.title('Interest Rate vs Total Bank Card Limit, Colored by Cluster')
plt.show()
plt.figure(figsize = (15,12))
sns.scatterplot(y='annual_inc_joint', x='bc_open_to_buy', hue='cluster', data=trim_df3, palette='Spectral')
plt.ylabel('Annual Income')
plt.xlabel('Bank Card Available Credit')
plt.title('Annual Income vs Available Bank Card Credit, Colored by Cluster')
plt.show()
Once again, available credit and income seem to be the variables driving cluster assignments.
xtra_trim_df3 = trim_df3.sample(2000, random_state=42)
temp_list3 = xtra_trim_df3['cluster']
xtra_trim_df3 = xtra_trim_df3.drop('cluster', 1)
predict3 = km3.fit_predict(xtra_trim_df3)
metrics.adjusted_rand_score(temp_list3, predict3)
xtra_trim_df3 = trim_df3.sample(2000, random_state=20)
temp_list3 = xtra_trim_df3['cluster']
xtra_trim_df3 = xtra_trim_df3.drop('cluster', 1)
predict3 = km3.fit_predict(xtra_trim_df3)
metrics.adjusted_rand_score(temp_list3, predict3)
xtra_trim_df3 = trim_df3.sample(2000, random_state=2)
temp_list3 = xtra_trim_df3['cluster']
xtra_trim_df3 = xtra_trim_df3.drop('cluster', 1)
predict3 = km3.fit_predict(xtra_trim_df3)
metrics.adjusted_rand_score(temp_list3, predict3)
The average ARI is 0.872 and the range is 0.120. Setting K=12 is clearly less stable than K=5.
K-Means, K=20¶
from sklearn.cluster import KMeans
km4 = KMeans(n_clusters=20, random_state=42).fit(trim_df4)
km4_labels = km4.labels_
km4_cluster_centers = km4.cluster_centers_
trim_df4.cluster.value_counts()
plt.figure(figsize = (15,12))
sns.scatterplot(y='int_rate', x='all_util', hue='cluster', data=trim_df4, palette='Spectral')
plt.ylabel('Interest Rate')
plt.xlabel('Utilization Rate')
plt.title('Interest Rate vs Utilization Rate, Colored by Cluster')
plt.show()
plt.figure(figsize = (15,12))
sns.scatterplot(y='int_rate', x='dti_joint', hue='cluster', data=trim_df4, palette='Spectral')
plt.ylabel('Interest Rate')
plt.xlabel('Debt to Income Ratio')
plt.title('Interest Rate vs DTI Ratio, Colored by Cluster')
plt.show()
plt.figure(figsize = (15,12))
sns.scatterplot(y='int_rate', x='bc_open_to_buy', hue='cluster', data=trim_df4, palette='Spectral')
plt.ylabel('Interest Rate')
plt.xlabel('Bank Card Available Credit')
plt.title('Interest Rate vs Available Bank Card Credit, Colored by Cluster')
plt.show()
plt.figure(figsize = (15,12))
sns.scatterplot(y='int_rate', x='mo_sin_old_rev_tl_op', hue='cluster', data=trim_df4, palette='Spectral')
plt.ylabel('Interest Rate')
plt.xlabel('Months Since Oldest Credit Line Opened')
plt.title('Interest Rate vs Months Since Oldest Credit Line Opened, Colored by Cluster')
plt.show()
plt.figure(figsize = (15,12))
sns.scatterplot(y='int_rate', x='total_bc_limit', hue='cluster', data=trim_df4, palette='Spectral')
plt.ylabel('Interest Rate')
plt.xlabel('Total Bank Card Limit')
plt.title('Interest Rate vs Total Bank Card Limit, Colored by Cluster')
plt.show()
plt.figure(figsize = (15,12))
sns.scatterplot(y='annual_inc_joint', x='bc_open_to_buy', hue='cluster', data=trim_df4, palette='Spectral')
plt.ylabel('Annual Income')
plt.xlabel('Bank Card Available Credit')
plt.title('Annual Income vs Available Bank Card Credit, Colored by Cluster')
plt.show()
As with most all the clustering techniques we've evaluated, available credit and income are the variables driving cluster assignment, although the dividing lines between clusters along these axes is less clear than in some of the earlier iterations.
xtra_trim_df4 = trim_df4.sample(2000, random_state=42)
temp_list4 = xtra_trim_df4['cluster']
xtra_trim_df4 = xtra_trim_df4.drop('cluster', 1)
predict4 = km4.fit_predict(xtra_trim_df4)
metrics.adjusted_rand_score(temp_list4, predict4)
xtra_trim_df4 = trim_df4.sample(2000, random_state=20)
temp_list4 = xtra_trim_df4['cluster']
xtra_trim_df4 = xtra_trim_df4.drop('cluster', 1)
predict4 = km4.fit_predict(xtra_trim_df4)
metrics.adjusted_rand_score(temp_list4, predict4)
xtra_trim_df4 = trim_df4.sample(2000, random_state=2)
temp_list4 = xtra_trim_df4['cluster']
xtra_trim_df4 = xtra_trim_df4.drop('cluster', 1)
predict4 = km4.fit_predict(xtra_trim_df4)
metrics.adjusted_rand_score(temp_list4, predict4)
The average ARI is 0.802 and the range is 0.205. This is definitely the weakest K-Means option evaluated.
Summary¶
From both the quantitative (Adjusted Rand Index scores) and qualitative (diagnostic plots) perspectives, employing a K-Means clustering algorithm with 5 clusters is the best option evaluated. Also, from the diagnostic plots we can tell that available credit and annual income of the borrower are the variables that really drive clustering. This tells us that if we want to further analyze or make some decisions about these loans that we should pay particular attention to these variables.
That being said, I feel that clustering is much less useful on this data set than modeling (i.e. regression modeling). With all of this work, all we've really learned is that available credit and annual income are probably the most important variables for differentiating one loan from another. However, with a regression model we could predict performance of loans or the expected interest rate for a given borrower profile and use this information to make investment decisions. In short, clustering helps us learn more about this data set, but only modeling can help us make decisions based on the data.